This study clarified the characteristics of workers who have smoothly shifted from office work to working from home under the COVID-19 pandemic, and investigated its impact on income, working hours and workers’ anxiety, using the employees’ longitudinal data collected by the internet survey in April and May 2020 in Japan. The results of the regression analyses showed that those who are university-graduated, regular employees, high earners, working in large companies, and working in companies which have well-human resource management had high possibility of switching to working from home under the pandemic, even after controlling for each job’s potentialities of working from home. This implied not only that those workers tended to occupy jobs which can be done from home, but also even if their jobs have the same potential to work from home, those who are less educated, non-regular employees and less earners had less choice to work from home. This study also investigated the impact of working from home on income, working hours and anxiety. To control for inverse causal effect, we conducted the instrumental variable methods. The analyses showed that working from home had significant effects of preventing from large earning loss and large decrease of working hours, but there was no significant effect on anxiety. In conclusion, it was shown that there was inequality of choices of working from home between workers’ characteristics under the pandemic, and also the implementation of working from home under the pandemic would produce another inequality of income and working hours.