Unluckily my problem of contact visibility is still there, but luckily I eventually probably found out the confusing aspect of this issue.
I would believe the Contact visibility of Odoo in a multi-companies setup is working.
The reason users still able to see contacts not belonged to their company because the 'Company' field I seen on my Odoo setup IS NOT the 'Company' required to distinguished between companies in Odoo.
Why I know?
When I create a contact via Contact Form, this contact is visible/invisible properly among companies.
However, when I import contacts from xlsx with the field 'Company' as I seen, these contacts visible to all users.
That means the 'Company' seen is not the same as the Odoo 'Company' required, (or an additional field required).
So what help?
I create a contact from Odoo Contact Form and export that contact from Action->Export and select required fields to export, there are many Company field for selection, will you know which 'Company' is the 'Company' required by Odoo security control?
You can restrict the allowed companies for each user under "Settings" > "Users" > "User Name". Under the tab "Access Rights" you can select which companies are allowed for this user. If a user is allowed multiple companies you can (de)activate companies left of the username in the top right corner. Also make sure that the contacts are assigned to the correct company. I hope this answers your question.