Solid-state energy converters, including thermoelectric, photovoltaic and photocatalytic devices, hold great promise of providing sustainable clean energy and addressing global challenges such as climate change and air/water pollution. One common feature of these devices is that their performance is largely determined by the transport and interaction processes of microscopic energy carriers, mainly electrons, phonons, and photons. It is being increasingly recognized that a thorough understanding of these processes is the key to the ultimate performance of a wide range of energy materials. In this talk, I will first report our recent development and application of DFT-based first-principles simulation tools to understand electron-phonon interactions in thermoelectric materials, as well as a novel three-pulse photoacoustic technique to directly measure phonon damping by electrons in silicon membranes. In the second half of the talk, I will introduce the ultrafast electron beam, which combines the femtosecond time resolution and nanometer spatial resolution, as an ideal probe to reach the single-mode-level detection of transport and interaction. As a demonstration, I will present visualization of photocarrier dynamics in hydrogenated amorphous silicon and black phosphorus using the scanning ultrafast electron microscope (SUEM), and discuss the findings that can only be revealed by ultrafast spatial-temporal imaging. To conclude, I will briefly discuss a few current ongoing projects at UCSB.