Please read the FAQ regarding xbox360 file format compatibility.  It may help you pick better options for your particular files than the general ones i offer below.

The solution is simple: Use Avidemux.

1. Install avidemux. To install it is simple. It’s in the Ubuntu repositories and I imagine you can also find it in other distros quite easily.
2. Open avidemux and open the mkv you want to convert. If prompted some garbage about 264 and safemode, just use safe mode and dont worry about it.
3. Select File -> Properties. This will tell you some info on the formats in use in your video file. Take this opportunity to identify what the xbox doesn’t like. When you are finished, click OK. If you want a second opinion, open up the folder with your mkv in it. Right click -> properties -> Audio/Video tab. This will also tell you the video and audio formats.
4. We obviously know that it wont play a video in a MKV container, so first thing to do is change the “Format” dropdown to say “MP4″ (you can, of course use AVI, but the majority of files I run into are h.264 and aac audio. For this combo, you want mp4…)
5. From the Properties menu, recall the video codec. H264 files show as “AVC1″ inside of avidemux. I’m sure theres a technical reason for this, but do you want to talk about it or watch your video?
   Most of the time, you can leave the video droptown in avidemux on “Copy” this is nice because it means that your processor wont be re-encoding the video. This saves you quality and time.
6. From the properties menu, recall the audio codec. If you have a video with AAC stereo audio, leave the dropdown on “copy”.
   This is where most of my files need some love. Many MKV’s have 5.1 surround audio tracks. This is great, but not for an xbox360. To mix the audio down to stereo, select AAC on the audio dropdown, then click “filters”. In the mixer dropdown, select “stereo”.
7. Click “save”. Avidemux will prompt you for a filename for the converted file. It does not default a file extension, so do yourself and your xbox a favor and add one yourself like “<videoname>.mp4″.

Once avidemux is finished with your file, it’s ready to go.

http://avidemux.sourceforge.net/