More than 60 per cent of respondents to the survey conducted in January said China concerned them, compared with 46 per cent in a similar poll three years earlier. The number worried about North Korea fell to about 53 per cent from around 65 per cent.
Asia's two largest economies are at loggerheads over uninhabited islands in the East China Sea, with ships and planes from both countries frequently criss-crossing near the disputed area. North Korea is developing atomic and ballistic missile technology, though it hasn't held a nuclear test since 2013.
"There is a lack of transparency in China's military and security policy, including about the budget," Defence Minister Gen Nakatani told reporters on Friday. "We want to continue to seek disclosure from China."
The two countries are set to hold security talks on March 19 in Tokyo, the first of their kind in four years.
The discussions come after Mr Abe met Chinese President Xi Jinping in Beijing in November, in the first bilateral summit between the countries since May 2012.
When asked in the survey about defence ties with countries other than the US, the proportion seeing benefits from military exchanges with China and South Korea fell by about a third.
Southeast Asia was the partner most commonly cited as useful to Japan.
Prime Minister Shinzo Abe is seeking to strengthen Japan's Self-Defense Forces, partly in response to China's modernization of its military.
While 59 per cent said the current size of Japan's armed forces was appropriate, a growing minority favor a military build-up. Almost 30 per cent said they wanted an expansion, up from 25 per cent in the previous poll and 14 per cent six years ago.
A record 71.5 per cent said they were interested in the SDF, and more than 92 per cent had a good impression of the armed forces.
The Cabinet Office interviewed 1,680 people between January 8-18. The survey was first conducted in 1969.
A Japan Air Self-Defense Force maintenance officer, salutes an F-15 Eagle pilot.