As asbestos-containing waste from building demolitions has to undergo special treatment prior to recycling or final disposal, the demolition of buildings and the sorting of waste materials are conducted after checking for asbestos content in the construction materials (according to ISO, EPA or JIS). However, as solid waste is usually a mixture, it is possible that asbestos-containing material (ACM) is present in construction and demolition waste (CDW) that is transported to an intermediate treatment facility for CDW. In addition, the presence of ACM in disaster waste cannot be avoided. Therefore, a rapid method for the determination of asbestos at an intermediate treatment facility for CDW is required. In this study, the separation efficiency and the sorting time of CDW particles by grouping by visual appearance (GVA) were determined. In the case that the separation efficiency by GVA in this study was equivalent to that by visual observation with a loupe (DVL) in a previous study, the reduction of sorting time by GVA was evaluated. Newton’s separation efficiency by GVA and recovery rate were equivalent to that by DVL for 5.1 cm2 observation. In this case, the sorting time by GVA was 1/7 of that by DVL. Therefore, sorting time could be shortened by GVA under the condition of equivalent separation efficiency. In order to reduce the sorting time per worker to less than 1 h/t, only CDWPs having particle size larger than 12 cm for GVA or 20 cm for DVL for 5.1 cm2 observation should be subjected to sorting. Aiming to avoid diffusion of asbestos-containing waste, the authors suggest that grouping by visual appearance as a primary sorting step is effective to reduce sorting time of CDW from disaster waste or unknown origin.