Wolfson was born in Vilna, Poland, in 1928 and immigrated to America at the age of 17 with his mother and young brother. He immediately went to work while sending his brother to yeshiva. In his 20s, Wolfson amassed a significant wealth through his investments in real estate.