Abstract: Medical reports generations especially in a third world countries can take unnecessary long hours. Efficient speech recognition technology trained under medical lexicon can greatly help not only the physicians but also the patients by reducing a lot of time. In order to build a robust speech recognition model that can not only differentiate and understand medical terminologies along with other generic terms and utterances but also grasps the nuances and structures of very different language.