Initial commit - Current state of vip-coordinator
This commit is contained in:
179
DOCKER_TROUBLESHOOTING.md
Normal file
179
DOCKER_TROUBLESHOOTING.md
Normal file
@@ -0,0 +1,179 @@
|
||||
# Docker Container Stopping Issues - Troubleshooting Guide
|
||||
|
||||
## 🚨 Issue Observed
|
||||
|
||||
During development, we encountered issues where Docker containers would hang during the stopping process, requiring forceful termination. This is concerning for production stability.
|
||||
|
||||
## 🔍 Current System Status
|
||||
|
||||
**✅ All containers are currently running properly:**
|
||||
- Backend: http://localhost:3000 (responding correctly)
|
||||
- Frontend: http://localhost:5173
|
||||
- Database: PostgreSQL on port 5432
|
||||
- Redis: Running on port 6379
|
||||
|
||||
**Docker Configuration:**
|
||||
- Storage Driver: overlay2
|
||||
- Logging Driver: json-file
|
||||
- Cgroup Driver: systemd
|
||||
- Cgroup Version: 2
|
||||
|
||||
## 🛠️ Potential Causes & Solutions
|
||||
|
||||
### 1. **Graceful Shutdown Issues**
|
||||
**Problem:** Applications not handling SIGTERM signals properly
|
||||
**Solution:** Ensure applications handle shutdown gracefully
|
||||
|
||||
**For Node.js apps (backend/frontend):**
|
||||
```javascript
|
||||
// Add to your main application file
|
||||
process.on('SIGTERM', () => {
|
||||
console.log('SIGTERM received, shutting down gracefully');
|
||||
server.close(() => {
|
||||
console.log('Process terminated');
|
||||
process.exit(0);
|
||||
});
|
||||
});
|
||||
|
||||
process.on('SIGINT', () => {
|
||||
console.log('SIGINT received, shutting down gracefully');
|
||||
server.close(() => {
|
||||
console.log('Process terminated');
|
||||
process.exit(0);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### 2. **Docker Compose Configuration**
|
||||
**Current issue:** Using obsolete `version` attribute
|
||||
**Solution:** Update docker-compose.dev.yml
|
||||
|
||||
```yaml
|
||||
# Remove this line:
|
||||
# version: '3.8'
|
||||
|
||||
# And ensure proper stop configuration:
|
||||
services:
|
||||
backend:
|
||||
stop_grace_period: 30s
|
||||
stop_signal: SIGTERM
|
||||
|
||||
frontend:
|
||||
stop_grace_period: 30s
|
||||
stop_signal: SIGTERM
|
||||
```
|
||||
|
||||
### 3. **Resource Constraints**
|
||||
**Problem:** Insufficient memory/CPU causing hanging
|
||||
**Solution:** Add resource limits
|
||||
|
||||
```yaml
|
||||
services:
|
||||
backend:
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 512M
|
||||
reservations:
|
||||
memory: 256M
|
||||
```
|
||||
|
||||
### 4. **Database Connection Handling**
|
||||
**Problem:** Open database connections preventing shutdown
|
||||
**Solution:** Ensure proper connection cleanup
|
||||
|
||||
```javascript
|
||||
// In your backend application
|
||||
process.on('SIGTERM', async () => {
|
||||
console.log('Closing database connections...');
|
||||
await database.close();
|
||||
await redis.quit();
|
||||
process.exit(0);
|
||||
});
|
||||
```
|
||||
|
||||
## 🔧 Immediate Fixes to Implement
|
||||
|
||||
### 1. Update Docker Compose File
|
||||
```bash
|
||||
cd /home/kyle/Desktop/vip-coordinator
|
||||
# Remove the version line and add stop configurations
|
||||
```
|
||||
|
||||
### 2. Add Graceful Shutdown to Backend
|
||||
```bash
|
||||
# Update backend/src/index.ts with proper signal handling
|
||||
```
|
||||
|
||||
### 3. Monitor Container Behavior
|
||||
```bash
|
||||
# Use these commands to monitor:
|
||||
docker-compose -f docker-compose.dev.yml logs --follow
|
||||
docker stats
|
||||
```
|
||||
|
||||
## 🚨 Emergency Commands
|
||||
|
||||
If containers hang during stopping:
|
||||
|
||||
```bash
|
||||
# Force stop all containers
|
||||
docker-compose -f docker-compose.dev.yml kill
|
||||
|
||||
# Remove stopped containers
|
||||
docker-compose -f docker-compose.dev.yml rm -f
|
||||
|
||||
# Clean up system
|
||||
docker system prune -f
|
||||
|
||||
# Restart fresh
|
||||
docker-compose -f docker-compose.dev.yml up -d
|
||||
```
|
||||
|
||||
## 📊 Monitoring Commands
|
||||
|
||||
```bash
|
||||
# Check container status
|
||||
docker-compose -f docker-compose.dev.yml ps
|
||||
|
||||
# Monitor logs in real-time
|
||||
docker-compose -f docker-compose.dev.yml logs -f backend
|
||||
|
||||
# Check resource usage
|
||||
docker stats
|
||||
|
||||
# Check for hanging processes
|
||||
docker-compose -f docker-compose.dev.yml top
|
||||
```
|
||||
|
||||
## 🎯 Prevention Strategies
|
||||
|
||||
1. **Regular Health Checks**
|
||||
- Implement health check endpoints
|
||||
- Monitor container resource usage
|
||||
- Set up automated restarts for failed containers
|
||||
|
||||
2. **Proper Signal Handling**
|
||||
- Ensure all applications handle SIGTERM/SIGINT
|
||||
- Implement graceful shutdown procedures
|
||||
- Close database connections properly
|
||||
|
||||
3. **Resource Management**
|
||||
- Set appropriate memory/CPU limits
|
||||
- Monitor disk space usage
|
||||
- Regular cleanup of unused images/containers
|
||||
|
||||
## 🔄 Current OAuth2 Status
|
||||
|
||||
**✅ OAuth2 is now working correctly:**
|
||||
- Simplified implementation without Passport.js
|
||||
- Proper domain configuration for bsa.madeamess.online
|
||||
- Environment variables correctly set
|
||||
- Backend responding to auth endpoints
|
||||
|
||||
**Next steps for OAuth2:**
|
||||
1. Update Google Cloud Console with redirect URI: `https://bsa.madeamess.online:3000/auth/google/callback`
|
||||
2. Test the full OAuth flow
|
||||
3. Integrate with frontend
|
||||
|
||||
The container stopping issues are separate from the OAuth2 functionality and should be addressed through the solutions above.
|
||||
Reference in New Issue
Block a user