There is a vast literature proposing non-Bayesian methods for making inferences incorporating auxiliary information such as population-level marginal moments. However, it is not feasible to apply these methods directly to latent variable models because the data augmentation approach, in which latent variables are treated as incidental parameters and then generated, is not developed. In this paper, we propose a Markov Chain Monte Carlo (MCMC) algorithm with data augmentation for latent variable models for cases in which we have both a sampled dataset and additional information such as population level moments. The resulting quasi-Bayesian inference with auxiliary information is very straightforwaed to implement, and consistency and asymptotic variance of the quasi-Bayesian posterior mean estimators from the MCMC outputs are shown in this paper. The proposed method is especially useful when the dataset is biased but we have an unbiased large sample for some variables or population marginal moments in which it is difficult to correctly specify the sample selection model. For illustrative purposes, we apply the proposed estimation method to generalized linear mixed models for biased data both in simulation studies and in real data analysis. The proposed method can be used to make inferences in non/semi-parametric latent variable models by incorporating the existing semi-parametric Bayesian algorithms such as the Blocked Gibbs sampler in the MCMC iteration.