Personalised pose estimation from single-plane moving fluoroscope images using deep convolutional neural networks.

Measuring joint kinematics is a key requirement for a plethora of biomechanical research and applications. While x-ray based systems avoid the soft-tissue artefacts arising in skin-based measurement systems, extracting the object's pose (translation and rotation) from the x-ray images is a time-consuming and expensive task. Based on about 106'000 annotated images of knee implants, collected over the last decade with our moving fluoroscope during activities of daily living, we trained a deep-learning model to automatically estimate the 6D poses for the femoral and tibial implant components. By pretraining a single stage of our architecture using renderings of the implant geometries, our approach offers personalised predictions of the implant poses, even for unseen subjects. Our approach predicted the pose of both implant components better than about 0.75 mm (in-plane translation), 25 mm (out-of-plane translation), and 2° (all Euler-angle rotations) over 50% of the test samples. When evaluating over 90% of test samples, which included heavy occlusions and low contrast images, translation performance was better than 1.5 mm (in-plane) and 30 mm (out-of-plane), while rotations were predicted better than 3-4°. Importantly, this approach now allows for pose estimation in a fully automated manner.

Introduction

Accurate measurement of joint kinematics is a key requirement for a variety of biomechanical and medical applications, such as investigating the effect of pathologies, the loading conditions and injury mechanism of joints, and the development of implants. The most common way of measuring skeletal kinematics is optical marker tracking, in which infrared cameras observe reflective markers glued to the subject’s skin and determine the marker positions through triangulation. Since the markers are attached to the skin, the markers’ movement differs from that of the underlying joint, thus leading to soft-tissue artefacts, which are inherent to all skin-based measurement systems [1, 2].

Contrary to such indirect approaches, x-ray methods measure the skeletal segments in a more direct manner, yielding a series of x-ray images of the joint during movement, from which the 3D kinematics can be estimated. Conventionally, this estimation has been performed by acquiring the 3D information of the target joint (e.g. using CT) and then manually adjusting the pose until the projection matches the x-ray image. This process takes about one minute per image for an experienced operator, or about five hours of manual matching work for each ten seconds of measurement. Because this process is so time-intensive, expensive, and operator-dependent [3] even for high-contrast structures such as joint replacements, there is considerable interest in a process to perform this 2D-3D pose estimation automatically.

While many approaches towards automation have been developed over recent years, the need for relatively accurate starting poses means that significant manual work is still required in practice [3-6]. With the rise of deep learning, pose estimation has attracted attention from the computer vision community, particularly for tasks such as industrial automation and robotics, and achieved impressive results in terms of performance and speed over conventional pose estimation methods. However, training deep learning methods requires large datasets with accurate pose annotations, which are extremely laborious and costly to collect for real images. Thus, most work to date has focused on everyday photographs and depth images on very specific and limited datasets [7], with only few studies exploring the medical imaging domain [8]. On top of the difficulty of acquiring a sufficient number of datasets, the strict data protection rules in many countries limit how data can be used and require special computational infrastructure to perform deep learning experiments on such data.

Over the last two decades, our research group has taken, registered, and annotated over 100’000 x-ray images of knee replacements during activities of daily living using our moving single-plane fluoroscope [9-11]. The high contrast offered by knee replacements and the standardized, high quality pose annotations of this unique dataset offers a ideal opportunity to apply deep-learning methods.

Contrary to most pose estimation tasks in computer vision, the implants’ 3D information is available for every measurement—typically as a surface file from the manufacturer—allowing the pose estimation algorithm to be personalised to each subject and their specific implant combinations. While various approaches for personalising deep-learning models have been proposed, most of these methods require additional manually annotated data for the personalisation process. To avoid the need for such additional annotation, it is possible instead to employ a personalisation process based on leveraging synthethic x-ray images created by rendering the known implant geometries [12, 13].

In this work, we have taken advantage of our unique dataset and the new LeoMed ETH computational cluster, which was specifically designed to facilitate deep learning experiments on medical data, to 1) extend current deep learning approaches to pose estimation of knee replacements from x-ray images and 2) personalise these approaches to each subject and their specific implants. The vision of this work was to create a fully automatic approach to registering x-ray images in a rapid and robust manner for all radio-opaque implants.

Materials and methods

Dataset

Our dataset consisted of about 106’000 x-ray images with manually annotated poses, and including unique 53 tibial/femoral component combinations, where different implant sizes were considered different implants. The images were collected over the last two decades at the Institute for Biomechanics at ETH Zurich during various clinical and biomechanical studies—using our moving fluoroscope to investigate joint/implant kinematics during activities of daily living, including walking, ramp descent, and stair descent, as well as knee bending activities [9-11].

For this work, the images were downscaled from 1000x1000 pixels (about 0.3 mm pixel) to 512x512 pixels to reduce memory and computational demands. Using the calibration information available for each measurement, we standardized the poses to a principal point at the center of the image intensifier and a focal distance of 0.97 m. The data was split into a training dataset from 49 of the unique tibial/femoral component combinations, and a test dataset of 4 combinations, which was used to estimate the performance of our approach on unseen subjects.

Network architectures

The Deep Auto-Match Network (DAMN) (Fig 1) is split into two parts: a) a “shadownet” network that extracts the implant shadows from the x-ray image and b) a “synthnet” network that estimates the poses from these shadows. The intuition behind this choice is that extracting the implant shadow from the x-ray is a general process that a human could do without information on the implant geometry, just by noting the strong attenuation of the implant components and the structure of the surrounding musculoskeletal tissue, such as the thigh and shank segments. However, the same shadow could imply completely different poses for different implants, so the pose estimation step clearly needs additional information on the implants’ geometries. By using rendered images of the known component geometries in varying poses, the “synthnet” learns which landmarks indicate what pose for a certain tibia/femur component combination—this process is similar to turning an unknown object to find that a certain edge is only visible for certain poses of the object. So, when this specific edge later appears in an image, the pose must be within the previously established range.

Fig 1

The shadownet extracts the implant shadow from the x-ray image, while the synthnet estimates the pose from this shadow for the known tibia/femur implant geometries.

Clearly, information on the implants’ geometries can help in extracting the shadow from the x-ray image—particularly for challenging images, which often include partial occlusion e.g. when the second leg passes during gait and decreases the image contrast. For such images, even a human would need additional information on how the shadow for this implant combination could appear. As a consequence, in a second experiment, we extended our architecture to iteratively consider the rendered implant shadows based on the estimated pose. From the second iteration onwards, the shadownet extracts the shadow not only from the x-ray image, but can also use the information from the rendered shadow. For the non-iterative versions of our approach and the first iteration of the iterative version, a black image was used for this rendered shadow.

The architecture for the shadownet was based on a Deeplab-v3 segmentation network with a Resnet-50 backbone [14]. The input layer was changed to accept 2 channels—the x-ray image, and the image of the rendered components in the estimated pose. The network architecture for the synthnet was a ResNeXt-50 model [15] (32 groups of width 4) with a 1-channel input layer and a fully connected output layer giving the 18 pose regression targets—3 translation components and a 6D representation of the rotation [16] for both the tibia and femur. We chose to estimate both components at the same time to allow each component to profit from the other’s likely position as well as from general information, such as features relating to surrounding structures, including the positions of the thigh and shank. We further replaced all of the batch normalisation layers with instance normalisation layers, which is better suited for the small batch sizes we use during training and our recursive architecture [17, 18].

Synthnet-training

First, we pretrain the synthnet on synthetically rendered, binarized (shadow) images of each individual femoral/tibial implant combination. Because the parameter space of the two implant poses is so high-dimensional, using purely random poses would lead to a lot of non-sensical situations, such as when the femur and tibia are impossibly overlapping or impossibly far apart considering the anatomical constraints, or would lead to poses that are irrelevant for activities of daily living (e.g. upside-down). To keep the model focussed on the task and the computational time low, it is thus important to focus on creating implant poses that are close to the ones arising in activities of daily living. Towards this goal, we perturb poses from our dataset of activities of daily living using Gaussian noise for each of the 18 pose components to create realistic poses close to the original kinematics.

For each implant component combination, we removed all entries of this combination in the dataset for training to avoid the synthnet overfitting on any of the poses it will later encounter during the training of the shadownet. To make the synthnet more resistant to occlusions, we implemented the CutOut [19] method with a probability of 0.3, and 4 squares of 128 pixels, corresponding to one quarter of the image size. The translation components of the pose were normalized over the dataset and the rotation components were transformed from Euler angles to a 6D representation [16], giving a total of 18 regression targets.

We used batch sizes of 4 samples, and ADAM as an optimiser (learning rate of 2 * 10−3, β values of 0.9 and 0.99, and no weight decay). As it is known that batch-size plays a key role during training [20], we accumulated the gradients of 32 samples before updating the weights during optimisation.

The pose-loss was given by a combination of translation losses L and rotation losses L for both the femur and the tibia poses: with being the target pose and being the predicted pose.

The translation losses L were weighted L1-losses with the in-plane directions (x,y) having a relative weight of 10 compared to the out-of-plane direction (z). This choice reflects the fact that out-of-plane translations are more difficult to estimate, as variations in this component lead only to small changes in the size of the implant components due to perspective. A Geodesic loss was used for the rotational loss L [16] with R the rotation matrix corresponding to the predicted pose of either femoral or tibial implant, and R the rotation matrix corresponding to the true pose.

Using this procedure, we first trained one synthnet to completion for about five days, and then froze the first quarter of the network before training the individualised networks for each of the 53 femoral/tibial implant combinations, which took about one day for each synthnet on a Nvidia GTX2080Ti GPU. Freezing the input layers of the synthnet ensured that the input features remained consistent between different synthnets, which was required as the synthnet part was swapped out during the training process of the shadownet in the next step.

Shadownet-training

To train the shadownet part of the DAMN, we loaded the corresponding pretrained synthnet for each batch. By freezing the weights of the synthnet, we ensured that the training procedure only affected the weights of the generic shadownet. The optimiser was stochastic gradient descent with a learning rate of 10−2 and no momentum. Notably, we chose not to use momentum during this step, because all samples from a certain tibia/femur combination were loaded after one another. While this process avoided the requirement to frequently swap the synthnet, momentum would build up as long as the synthnet remained the same, but could be completely inappropriate as soon as the synthnet was switched to the next tibia/femur combination. The batch size for this training step was set to 2 samples for the non-iterative version and 1 sample for the iterative version of our approach. Similar to the training of the synthnet, we accumulated the gradients of 16 samples before updating the weights during stochastic gradient descent.

The loss function for this training step was given by the combination of the pose loss L (Eq 1) and a Lovasz-Hinge loss, which is a smooth extension of the Jaccard index [21, 22], as the segmentation loss L. Here, p and s are the true pose and true shadow (acquired by rendering the implants in the true pose p), while s describes the shadow predicted by the shadownet, p is the pose predicted by the synthnet. Further, w = 10 is a weight factor to make the two loss components comparable in magnitude. Training of the shadownet took about five days on a Nvidia GTX2080 Ti GPU, with improvements slowing down considerably after about three days. Overall, the training of the complete, personalised DAMN model took about 63 GPU days, which we parallelised using multiple GPUs. After the initial training, any new subject will only require a new personalised synthnet, which can be trained in 1 day using transfer-learning.

Results

Figs 2–5 show the performance of the individualised non-iterative and iterative approaches on the testset, re-emphasizing that no data from subjects in the testset was used during training. For 50% of all images in the 4 subject test dataset, the non-iterative approach predicted the pose of both implants better than about 0.75 mm for in-plane translations and 25 mm for out-of-plane translations, while the iterative approach improved the out-of-plane predictions to below 10 mm. For 90% of the testset, the non-iterative approach achieved 1.5 mm and 30 mm (in-plane, out-of-plane) and the iterative approach achieved 2.5 mm and 25 mm. The predicted rotations (presented as Euler angles for interpretability instead of the internal 6D representation) differed by less than about 2° from the ground truth over 50% of the testset for the non-iterative and less than 1.5° for the iterative version. Over 90% of the samples, both iterative and non-iterative versions predicted all rotations better than 3−4°.

Fig 2

Translation performance for the individualised non-iterative DAMN on the test-set.

Fig 5

Rotation (right) performance for the individualised 2-iteration DAMN on the test-set.

Discussion

This work presented an automated method to perform 2D-3D pose estimation from fluoroscopic images based on a deep-learning approach that is individualised to each subject and their implants. For a large part of the test-set, in-plane translations were predicted to within 0.75 mm—typically for clean x-ray images with high contrast and no occlusions. Predictions were made in a fully-automated manner, in less than 10 s for 100 images, saving about 90 min required for manual annotation.

Our approach struggled for images in which the implant was occluded, as frequently happens e.g. during normal gait when the legs pass each other and decrease the contrast. Notably, such images are challenging even for humans, decreasing the quality of manual poses used as ground truths in this work. Some subjects even had two knee replacements, which can be in the image at the same time or overlap; while the algorithm can properly identify the implant to be registered (the one closer to the image intensifier), overlapping leads to a distorted shadow segmentation. This is a classic case of distribution mismatch, because the synthnet is trained using ideal (rendered) shadows while the shadownet might not be able to provide those in all cases. For such difficult images, the in-plane translation performance dropped to about 1.5 mm. Overall, the in-plane performance is sufficiently high for most clinical and research applications on implant development and kinematics, but visual inspection of the images together with the predicted pose is advised to find major mis-predictions. Alternatively, the predicted pose could be used as the starting pose for a pose refinement optimisation [3, 4, 6], and convergence could be used as an indicator for successful pose estimation.

Even though the CutOut-method [19] used during training of the synthnet should confer some resilience to occlusions, extracting a shadow from an occluded image without any additional information simply remains challenging. Consequently, we proposed an extension to our architecture to iteratively feed the rendered implant shadows back into the shadownet, based on the estimated pose Fig 1. Suprisingly, the iterative version of our approach, which tries to provide some approximate information on how the object’s full shadow could look like, did not improve predictions on such harder images, but sometimes even worsened the registration performance. The reason for this result could be that the recursive networks are harder to train and require more computational resources due to their increased depth. Even using our dedicated computational cluster, we were still only able to train the network with 1 additional iteration, which is unlikely to be sufficient for the network to learn a new iterative process—particularly for these challenging images for which the predicted shadow and pose after the first iteration are still poor. To evaluate the true potential of the iterative approach, further investigation with a higher number of iterations will therefore be needed, most likely requiring a method to distribute the model across multiple GPUs e.g as described for volumetric data in [23].

Compared to the excellent predictions for in-plane translation, out-of-plane translations were much harder to estimate, predominantly because a change in the out-of-plane position only leads to a comparatively small change in the apparent size of the implant due to perspective. Consequently, we used a lower weight in the loss function for the out-of-plane translation, which makes the algorithm focus less on this component and resulted in a lower out-of-plane performance of 25 mm. Again, surprisingly, the iterative version of our approach improved this out-of-plane prediction to about 10 mm. As discussed above, training on networks with a higher number of iterations will be necessary to study this effect, and establish the trade-off between performance improvement and computational requirements.

Performance in pose estimation tasks is fundamentally limited by the image resolution and the quality of the 3D implant information. For example, the in-plane translation performance of our approach is about 0.75 mm, which corresponds to about one to two pixels for the downsampled 512px images. Considering this resolution limit, it is likely that the performance of our approach could be further improved by using the original 1000px images. However, use of these larger images substantially increases the size of the model and thus the required memory and training time by a factor of four. Considering the substantial training time (for a Nvidia GTX 2080 Ti: 5 days for the initial synthnet, 1 day for each implant combination, 5 days for the shadownet) this complexity increase seems infeasible impractical without making considerable improvements to either the model or the training process—particularly during development (e.g. dealing with occlusions) where rapid prototyping and evaluation is necessary.

Rather than using high resolution images in the deep-learning model directly, the predicted pose from a lower-resolution model could be used as the starting pose for a pose-refinement algorithm. Such pose-refinement algorithms have reported performances comparable to our results, when applied to high resolution images as long as they have been provided with good starting poses [3, 4, 6]. Since most of these algorithms work iteratively, one could save the one-time cost to train our deep-learning model on high-resolution images, but would pay with substantially longer times of multiple minutes per image for pose inference as compared to the mere seconds required for the deep-learning model.

One core limitation of this study is that manually annotated data are considered the ground truth. Even though a pose optimiser algorithm was applied after manual pose estimation to reduce inter-operator variability, the x-ray measurement system still had a limited resolution, and was subject to various distortion effects stemming from the experimental setup. Our results can thus only be interpreted in the context of replacing the manual pose estimation, and can only be considered as approximations with respect to the real poses. Notably, most other studies that investigated performance also suffered from this limitation, even though some aspects can be addressed through manufactured objects with known geometries [3]. On such objects, manual pose estimation achieved an in-plane performance between 0.3 mm to 0.6 mm and 7 mm out-of-plane, and predicted rotations to about 0.3 deg to 1.0 deg. However, these results are derived from relative poses, so no conclusions can be drawn about absolute poses. Furthermore, the calibration object was a simple geometric structure with good contrast, so the performance of manual pose estimation is likely to be worse for the more challenging implants, low contrast images, or images with occlusions. To assess the performance of various pose estimation approaches with respect to reality, future studies should acquire the true pose of more realistic objects and more realistic poses through an independent and substantially more accurate means, such as mechanical precision devices.

Our approach requires that the 3D information of the target object is accurately known—if the algorithm does not know what do look for, how should it be able to find and estimate its pose? While we had no way of measuring the implant geometries directly, manufacturing discrepancies from the implant designs (which were available) are assumed to be negligible compared to other error sources. If the 3D geometry of the target object is uncertain (e.g. when estimating the pose of natural bone, which is often acquired by CT, which has a limited resolution), this effect will be more pronounced and should be investigated. Here, different discrepancies (scale, loss of detail, slight geometric changes of one part of the outline) could be investigated with respect to their effect on the estimation result and compared to the results of the gold standard using the modified geometry. While this process seems prohibitively time- and cost-intensive when using manual registration as gold standard, an automated approach using mechanical precision devices could provide a practical implementation.

For the out-of-plane translations, the information available from single-plane fluoroscopic images is inherently limited, which has led various research groups to develop dual-plane fluoroscopes [5, 24, 25]. Our pose estimation approach can directly be applied to dual-plane images by simply training the synthnet on two synthetic x-ray shadows instead of one (taking into account the relative angles and positions of the dual-plane setup). Similarly, the overall DAMN would take one additional image as an input, with no other changes required. If a trained single-plane model is already available, transfer-learning could be used to acquire a dual-plane model, requiring only minimal annotated dual-plane data.

Our approach is designed to estimate the poses frame-by-frame. While this decision allows our method to work independently of frame rate or task performed, the algorithm remains ignorant of valuable time-information (where was the object one or two frames previously or subsequently?) and the allowed movement space for a certain task (how does the joint angle typically vary with time for healthy subjects?). In principle, a variety of deep learning approaches exist to deal with time-series and video-data, but would require exorbitant computational facilities as well as training different models for each device, frame rate, or task. A recent study proposed an interesting alternative, and trained an auto-encoder to learn the possible movement space of the joint for a certain task. Then, approximate poses were improved by a Kalman-filter that considered both the time-information over multiple frames but also the kinematic understanding learned by the auto-encoder [26].

Our approach can further be extended to estimate the poses of natural bone instead of implants. Assuming that the bone geometry is known (e.g. from CT), our approach can be used with no further modifications. Indeed, for certain applications, our approach of a segmentation and a personalized regression network can also be used to estimate poses from CT-data, thus opening opportunities in other fields using CT or μ-CT, such as research on porous media [27-29]. While our approach is easily extensible to deal with volumetric data (instead of 2D images) by using standard 3D-networks, the main limitation remains the requirement that the target object’s geometry must be known to personalise the regression network. Because implants have high absorption and thus appear as nearly uniform shapes in the x-ray image, our approach binarizes the segmentation and regression networks. For estimating the poses of structures with lower absorption that show more internal structure on x-ray images (e.g. bone), one could replace our binary segmentation and regression networks with greyscale versions to consider this additional information. However, it is likely that this version would be more susceptible to noise or occlusion effects, so it remains to be investigated whether a binarized or grey-scale approach is advantageous for estimating the poses of natural bone.

Conclusion

We presented a deep-learning based approach to estimate joint poses from fluoroscopic images, which is personalised to each subject without any additional manual pose registration. Depending on the required performance, our approach can be used directly to predict poses to about 0.75 mm (in-plane translation) and 2° (all Euler-angle rotations), and can easily be extended to natural bone, multi-plane fluoroscopy, or even CT, assuming that the target object’s geometry is known. Alternatively, our approach can be combined with pose refinement algorithms [3-6] by providing approximate starting poses in a fully automated manner, thus avoiding the laborious and expensive task of manually estimating the poses for each image.

9 Mar 2022

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Jyotismita Chaki, PhD

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please update your submission to use the PLOS LaTeX template. The template and more information on our requirements for LaTeX submissions can be found at http://journals.plos.org/plosone/s/latex.

3. Thank you for stating the following in the Acknowledgments Section of your manuscript:

This work was supported by Innosuisse (50579.1 IP-LS). The study sponsor

had no involvement in study design, in the collection, analysis and interpretation

of data, in the writing of the manuscript, and in the decision to submit the

manuscript for publication.

Please note that funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:

This work was supported by Innosuisse (50579.1 IP-LS, https://www.innosuisse.ch). The study sponsor had no involvement in study design, in the collection, analysis and interpretation of data, in the writing of the manuscript, and in the decision to submit the manuscript for publication.

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

4. Please note that in order to use the direct billing option the corresponding author must be affiliated with the chosen institute. Please either amend your manuscript to change the affiliation or corresponding author, or email us at plosone@plos.org with a request to remove this option.

5. We note that you have stated that you will provide repository information for your data at acceptance. Should your manuscript be accepted for publication, we will hold it until you provide the relevant accession numbers or DOIs necessary to access your data. If you wish to make changes to your Data Availability statement, please describe these changes in your cover letter and we will update your Data Availability statement to reflect the information you provide.

6. Please remove your figures from within your manuscript file, leaving only the individual TIFF/EPS image files, uploaded separately.  These will be automatically included in the reviewers’ PDF.

Additional Editor Comments:

Kindly modify the manuscript as per the reviewer comments and resubmit

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The paper is well-motivated and the introduction of the Deep Auto-Match Network (DAMN) is novel. However, there are some concerns that should be addressed:

1. DAMN consists of two parts and contains various loss functions, and it is recommended to perform ablation experiments to show the role of each part. More explanation and intuition on DAMN are needed.

2. It is recommended that the time complexity should be discussed and analyzed in detail. In addition, the parameters of the DAMN should also be analyzed numerically.

3. Since some data is difficult to obtain, it is worth considering whether the performance can be improved by generating data. For example, some work similar to [1] may have some inspiration.

[1] Gong, An, Xinjie Yao, and Wei Lin. "Dermoscopy image classification based on StyleGANs and decision fusion." Ieee Access 8 (2020): 70640-70650.

4. There are several works on the combination of deep learning and medicine, and it is suggested to compare with some state-of-the-art methods. In addition, this work discussed x-ray CT and its strengths and disadvantages, it is time-intensive, expensive, and operator-dependent. In fact, in the field of porous media/materials, the same problem exists with the use of X-CT, see, Fractals-Complex geometry patterns and scaling in nature and society, 2018, 26(2): 1840003. https://doi.org/10.1142/S0218348X18400030, and Physics of Fluids, 2021, 33(3): 032013. https://doi.org/10.1063/5.0042606. It is suggested that the authors refer to these recent works to strengthen the technical context of this ms, support the point of view about X-CT and attract more readers.

Reviewer #2: Paper deals with hot topic. It has scientific novelty and great practical value.

Paper has a logical structure all necessary sections. The paper is technically sound. The experimental section is good.

The proposed approach is logical, results are clear.

Suggestions:

(1) Can authors explain more about 6D representation of predicted rotation and automated method to perform 2D-3D pose estimation?

(2) How the quality of the 3D implant information effects performance in pose estimation task? Justify.

(3) In common we uses high resolution images in deep learning algorithms. What is your approach? If you are not using high resolution images then why?

(4) If it is not restricted in journal norms then one Conclusion Section will help interested authors to attract readers to go read this research work.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

2 May 2022

We would like to thank the reviewers for their critical feedback, which we have included and have helped to improve our manuscript. Please see the attached file "Response to Reviewers" for a detailed point-by-point response, including the relevant modifications for your convenience.

Submitted filename: Response to Reviewers2.docx

Click here for additional data file.

14 Jun 2022

Personalised pose estimation from single-plane moving fluoroscope images using deep convolutional neural networks

PONE-D-21-36228R1

Dear Dr. Vogl,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Jyotismita Chaki, PhD

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

I am happy to inform you that reviewers are satisfied with the revised manuscript and thus I am provisionally accepting the manuscript for publication.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

**********

16 Jun 2022

PONE-D-21-36228R1

Personalised pose estimation from single-plane moving fluoroscope images using deep convolutional neural networks

Dear Dr. Vogl:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Jyotismita Chaki

Academic Editor

PLOS ONE