Brumby made the proposal in a 10-point national health reform plan unveiled at a health conference in Melbourne on Tuesday.
He also called on the federal government to commit an extra $1 billion to the five-year Australian Health Care Agreements (AHCA), which are due to be renegotiated at the end of this month.
Brumby said the nation's major hospitals should have at least one Medicare-funded GP or an on-site GP clinic to deal with patients who would otherwise not require hospital treatment.
He said non-emergency cases were a drain on emergency departments, particularly on weekends and after hours, costing the state about $50 million a year.
"That's in Victoria, multiply that Australia-wide it's $200 million a year, over the course of an AHCA you're talking about close to a billion dollars of money that could be better spent and that's what the proposition is all about," Brumby told reporters.
"We believe we can provide better services for families and better value for money by having GPs based in those hospital outpatient departments.
"It's common sense, it would end the blame game, it's more efficient, it's better value for money and it would be better for parents and families."
Brumby said cost shifting currently prevented GP services at hospitals because the states paid for outpatients services while bulk-billing was funded by the commonwealth.
Health funding was previously split equally between state and federal governments but the commonwealth now only funded 40 per cent, Brumby said.
He said an extra $1 billion a year in national health funding would make up for the 10 per cent funding deficit by the federal government.
The reforms also call for a roll out of more super clinics - which provide specialist medical care for outpatients - across high priority areas and an expansion of the Medicare Benefits Schedule to boost the health care workforce.
Victorian opposition health spokeswoman Helen Shardey slammed the proposal, saying it was another example of the state government grabbing at straws to address the fact it was not managing the state's emergency departments properly and not providing enough hospital beds.
"This (is) an absolute admission that this government is totally failing in health," she said.
"There are no new ideas, there's no vision, all this is a grab for more money from the commonwealth from a government that is not performing in health."
