180 lines
4.3 KiB
Markdown
180 lines
4.3 KiB
Markdown
# Docker Container Stopping Issues - Troubleshooting Guide
|
|
|
|
## 🚨 Issue Observed
|
|
|
|
During development, we encountered issues where Docker containers would hang during the stopping process, requiring forceful termination. This is concerning for production stability.
|
|
|
|
## 🔍 Current System Status
|
|
|
|
**✅ All containers are currently running properly:**
|
|
- Backend: http://localhost:3000 (responding correctly)
|
|
- Frontend: http://localhost:5173
|
|
- Database: PostgreSQL on port 5432
|
|
- Redis: Running on port 6379
|
|
|
|
**Docker Configuration:**
|
|
- Storage Driver: overlay2
|
|
- Logging Driver: json-file
|
|
- Cgroup Driver: systemd
|
|
- Cgroup Version: 2
|
|
|
|
## 🛠️ Potential Causes & Solutions
|
|
|
|
### 1. **Graceful Shutdown Issues**
|
|
**Problem:** Applications not handling SIGTERM signals properly
|
|
**Solution:** Ensure applications handle shutdown gracefully
|
|
|
|
**For Node.js apps (backend/frontend):**
|
|
```javascript
|
|
// Add to your main application file
|
|
process.on('SIGTERM', () => {
|
|
console.log('SIGTERM received, shutting down gracefully');
|
|
server.close(() => {
|
|
console.log('Process terminated');
|
|
process.exit(0);
|
|
});
|
|
});
|
|
|
|
process.on('SIGINT', () => {
|
|
console.log('SIGINT received, shutting down gracefully');
|
|
server.close(() => {
|
|
console.log('Process terminated');
|
|
process.exit(0);
|
|
});
|
|
});
|
|
```
|
|
|
|
### 2. **Docker Compose Configuration**
|
|
**Current issue:** Using obsolete `version` attribute
|
|
**Solution:** Update docker-compose.dev.yml
|
|
|
|
```yaml
|
|
# Remove this line:
|
|
# version: '3.8'
|
|
|
|
# And ensure proper stop configuration:
|
|
services:
|
|
backend:
|
|
stop_grace_period: 30s
|
|
stop_signal: SIGTERM
|
|
|
|
frontend:
|
|
stop_grace_period: 30s
|
|
stop_signal: SIGTERM
|
|
```
|
|
|
|
### 3. **Resource Constraints**
|
|
**Problem:** Insufficient memory/CPU causing hanging
|
|
**Solution:** Add resource limits
|
|
|
|
```yaml
|
|
services:
|
|
backend:
|
|
deploy:
|
|
resources:
|
|
limits:
|
|
memory: 512M
|
|
reservations:
|
|
memory: 256M
|
|
```
|
|
|
|
### 4. **Database Connection Handling**
|
|
**Problem:** Open database connections preventing shutdown
|
|
**Solution:** Ensure proper connection cleanup
|
|
|
|
```javascript
|
|
// In your backend application
|
|
process.on('SIGTERM', async () => {
|
|
console.log('Closing database connections...');
|
|
await database.close();
|
|
await redis.quit();
|
|
process.exit(0);
|
|
});
|
|
```
|
|
|
|
## 🔧 Immediate Fixes to Implement
|
|
|
|
### 1. Update Docker Compose File
|
|
```bash
|
|
cd /home/kyle/Desktop/vip-coordinator
|
|
# Remove the version line and add stop configurations
|
|
```
|
|
|
|
### 2. Add Graceful Shutdown to Backend
|
|
```bash
|
|
# Update backend/src/index.ts with proper signal handling
|
|
```
|
|
|
|
### 3. Monitor Container Behavior
|
|
```bash
|
|
# Use these commands to monitor:
|
|
docker-compose -f docker-compose.dev.yml logs --follow
|
|
docker stats
|
|
```
|
|
|
|
## 🚨 Emergency Commands
|
|
|
|
If containers hang during stopping:
|
|
|
|
```bash
|
|
# Force stop all containers
|
|
docker-compose -f docker-compose.dev.yml kill
|
|
|
|
# Remove stopped containers
|
|
docker-compose -f docker-compose.dev.yml rm -f
|
|
|
|
# Clean up system
|
|
docker system prune -f
|
|
|
|
# Restart fresh
|
|
docker-compose -f docker-compose.dev.yml up -d
|
|
```
|
|
|
|
## 📊 Monitoring Commands
|
|
|
|
```bash
|
|
# Check container status
|
|
docker-compose -f docker-compose.dev.yml ps
|
|
|
|
# Monitor logs in real-time
|
|
docker-compose -f docker-compose.dev.yml logs -f backend
|
|
|
|
# Check resource usage
|
|
docker stats
|
|
|
|
# Check for hanging processes
|
|
docker-compose -f docker-compose.dev.yml top
|
|
```
|
|
|
|
## 🎯 Prevention Strategies
|
|
|
|
1. **Regular Health Checks**
|
|
- Implement health check endpoints
|
|
- Monitor container resource usage
|
|
- Set up automated restarts for failed containers
|
|
|
|
2. **Proper Signal Handling**
|
|
- Ensure all applications handle SIGTERM/SIGINT
|
|
- Implement graceful shutdown procedures
|
|
- Close database connections properly
|
|
|
|
3. **Resource Management**
|
|
- Set appropriate memory/CPU limits
|
|
- Monitor disk space usage
|
|
- Regular cleanup of unused images/containers
|
|
|
|
## 🔄 Current OAuth2 Status
|
|
|
|
**✅ OAuth2 is now working correctly:**
|
|
- Simplified implementation without Passport.js
|
|
- Proper domain configuration for bsa.madeamess.online
|
|
- Environment variables correctly set
|
|
- Backend responding to auth endpoints
|
|
|
|
**Next steps for OAuth2:**
|
|
1. Update Google Cloud Console with redirect URI: `https://bsa.madeamess.online:3000/auth/google/callback`
|
|
2. Test the full OAuth flow
|
|
3. Integrate with frontend
|
|
|
|
The container stopping issues are separate from the OAuth2 functionality and should be addressed through the solutions above.
|