Building vector extraction from aerial images is a challenge in many applications, especially location-based services. In recent years, different deep-learning techniques have improved the accuracy of building extraction. We propose a framework (JuliMa-Net) for building vector extraction from aerial images without boundary regularization or vertex sampling. The wireframe generated by our framework shows more detail about the building’s structure than its footprints. We initially selected three pretrained networks of Mask R-convolutional neural network, line, and junction detection. To improve the performance of the primary junction detection network, unique sets of decoders were designed. The smaller training set was used to fine-tune all three networks simultaneously. In addition, after applying proper processing on the obtained building masks, the outputs of the other two networks—the detected lines and junctions—were precisely selected and combined. The result of this process is a significant reduction in false detections and an increase in precision of 96%. Additionally, the final processing adds precise lines to the wireframes after combining junctions and lines. This improves the recall value.